Direct Mass Limits for Chiral Fourth-Generation Quarks in All Mixing Scenarios 
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Present limits on chiral fourth-generation quark masses ray and mj/ are broadly generalized and 
strengthened by combining both t' and b' decays and considering the full range of t' and h' flavor- 
mixing scenarios (with the lighter generations). Various characteristic mass-splitting choices are 
considered. With m^/ > ray we flnd that CDF limits on the h' mass vary by no more than 10-20% 
with any choice of flavor-mixing, while for the t' mass, we typically flnd stronger bounds, in some 
cases up to rati > 430 GeV. For ray > rat' we find my > 380 — 430 GeV, depending on the 
flavor-mixing and the size of the rat' — ray mass spfltting. 



PACS numbers: 

Despite many constraints by precision electroweak 
data, the number of simple Standard Model (SM) quark 
generations has not been definitively established. Recent 
searches by the CDF Collaboration placing lower limits 
on the masses of such objects [1, 2] have been construed 
to leave open significant portions of the theoretical land- 
scape that include a light fourth generation [3-7] . In this 
letter, we offer a treatment of the published data that 
fully probes the robustness of these limits and applies 
them to heavy quark decay modes for which there are 
no present direct limits. By considering all scenarios of 
mixing among four SM generations, we demonstrate that 
present data exclude a chiral fourth generation quark 
with a mass up to nearly 300 GeV, independent of the 
flavor-mixing assumptions. In so doing we establish a 
methodology for interpretation of future mass limits. 

Recent searches for direct production of fourth genera- 
tion quarks, denoted t' and b' for the up- and down-type, 
found nif > 335 GeV [1] and mi,' > 338 GeV [2], assum- 
ing B{t' -^ W{q = d, s, b}) = 100% and B{b' -> Wt) = 
100% respectively. Each assumes that an individual con- 
tribution from one flavor of fourth-generation quark com- 
prises the entire signal. These searches have been inter- 
preted under the assumptions of mt' — rrib' < Mw and 
negligible mixing of the {t\ b') states with the two lightest 
quark generations. To account for electroweak precision 
data [8] and flavor data [9] such conditions are typically 
required for SM extensions with four quark generations 
and one Higgs doublet; however, more exotic scenarios 
have also been proposed [5-7, 10] . 

This subtle theoretical landscape demands a broader 
view: there is no uniquely interesting set of assumptions 
under which experimental data should be interpreted. 
From the experimental view point, choosing a simple set 
of assumptions allows straightforward, if narrow, inter- 
pretation of results. Yet these interpretations may be 
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extended beyond the narrow mixing assumptions, and 
even applied to previously unexplored decay modes. In 
this article we consider the possible 4th generation mass 
spectrum and flavor-mixing space broadly to show that 
data from the CDF searches set direct limits for arbitrary 
mixing values and for both cases of [mj/ — TOft/| > Mw- 
In particular, we present results for the cases ttt-c > my 
and mf,' > TOf, and both modest (50 GeV) and large 
(100 GeV) mass splitting. 

CDF analyzed two independent samples: (1) a collec- 
tion of events containing a single lepton and at least four 
jets, denoted as the i + ij sample [1], and (2) a collection 
of events containing two same-charge leptons, two jets 
(one with a flavor tag) and evidence of neutrinos (miss- 
ing transverse energy), denoted as the i^i^jb^T sample 
[2]. We describe them here. 

In 4.6 fb"^ of data, CDF searched for t' -^ W{q = 
d, s,b} decays in the mode 



PP 



t't' ^{W -^ lv)q{W -^ qq)q 



by requiring a single lepton and at least four jets [1]. 
The data were analyzed by reconstructing the invariant 
mass of the candidate t' and measuring the total energy 
in the event. The event selection used the four jets of 
highest transverse energy in the event, but did not require 
a flavor signature (6-tag) on any of the jets, making it 
generally sensitive to t' -^ W{q — d,s,b}. Assuming 
B{t' ^ W{q = d, s, b}) ^ 100%, CDF found mj > 335 
GeV [1]. 

The mass reconstruction used minimum-likelihood fit- 
ting methods that depend upon the particular spectrum 
of final state components. If, for example, one half of an 
event decayed as t' — >• W{b' — ;■ Wq), giving a WWqWq 
topology, it might satisfy the £ + 4j selection criteria, but 
the reconstructed mass distribution for such events would 
be significantly modified by the additional W. Thus, 
the results cannot be trivially applied to topologies other 
than WqWq. We therefore apply the £ + Aj results ex- 
clusively to WqWq processes. 

CDF also searched for b' — >■ Wt decays in 2.7 fb^ of 
data, in the same-charge lepton mode: 



PP 



b'b' -^ WtWt 



WWbWWb 



by requiring two same-charge leptons, at least two jets 
(at least one with a 6-tag), and missing transverse en- 
ergy of at least 20 GeV [2]. Given the small back- 
grounds, multiple neutrinos and large jet multiplicity in 
the sample, CDF did not reconstruct the h' mass, but in- 
stead fit the observed jet multiplicity to signal and back- 
ground templates generated from simulations. Assuming 
B{h' -> Wt) = 100%, CDF found m'j, > 338 GeV [2]. 

The £^£^jb^T analysis did not use final-state depen- 
dent fits, thus results are process-independent and may 
be applied to any process producing the £ £ jb^T sig- 
nal. For example, t' -^ Wb' -^ WWt -^ WWWb decays 
would produce a six-VF, two-6 signature, with higher jet 
multiplicity and larger acceptance to the £^£^jb^T sam- 
ple than the simple ioui-W, two-6 signature. In this anal- 
ysis, we therefore apply the £'^£^jb^T results inclusively 
to processes resulting in at least four W bosons and two 
b quarks. 

The £ + Aj and £^£^jb^T data samples are comple- 
mentary. In the case that the fourth generation quarks 
decay exclusively at tree level through the charged cur- 
rent electroweak interaction (assured for a chiral fourth 
generation), the two searches can be minimally under- 
stood to probe two corners of a two-dimensional interval 
in branching fraction space. In particular, for the case 
where the t' is heavier than the b' the topologies of b' and 
t' decays are determined by four branching fractions, two 
of which are independent: 
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B{t'^Wb') = l-B{t' 
B{b'^Wt) = l-B{b' 



W{q^d,s,b}) 
W{q = u,c}) 



as shown in Fig. 1. In this representation, the £^£^jb^T 
and £ + 4j analyses probe complementary regions (see 
Fig. 1). 
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FIG. 1: The flavor-mixing intervals overlaid with a table of 
the processes contributing to the axis vertices. 

We consider the implications of the CDF data to vari- 
ous two-flavor (t' and b') scenarios, characterized by the 
t' — b' mass splitting and flavor-mixing rates. To extend 
the interpretations of the published results, we use the 
relationship among event yield, cross section and accep- 
tance to interpret the observed yield limits under the 
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FIG. 2: (a) Limits for the combined hypothesis t' — >■ Wb' and 
b' — > Wt derived from the I'^l'^jb^r data. Also plotted is 
the previous CDF limit for the individual b' case, (b) Limits 
on 6' mass from the combined £ £ jbpr and £ + Aj data, as 
a function of branching fractions B{t' — >■ Wb') = 1 — B{t' -^ 
Wq) [q = d, s, b] and B(b' ^ Wt) ^ 1 - B{b' -^ Wq) [q = u, c] 
for the case rrit' = my + 100 GeV (c) Same as in (b) but for 
rriti — nil,! + 50 GeV. (d) Limits on t' mass from £ + Aj data 
as a function of BR(6' — > W{q = u, c}), in the inverted mass 
splitting case m^/ > rriti . 



varying assumptions. This requires parametrization of 
the variation in relative acceptance due to modification 
of the signal-source model. 

First, we consider the £ £ jb^T sample, interpreted 
under the mass splitting assumption m^ > to;,'. In the 
original £^£^jb^T analysis, the event yield was assumed 
to come from an individual b'. Here, we interpret this 
yield under a model with both t' and b' contributions, 
where the decay modes of interest are 

b' ^ Wt^ WWb 

t' ^ Wb' -^ WWt -^ WWWb 

which corresponds to the boundary case (1,1) in branch- 
ing fraction space. The t' -^ WWWb mode has no prior 
direct limit despite having a similar signature with larger 
acceptance in the £^£^jb^T dataset due to the two ad- 
ditional Ws in the intermediate decay chain. Indeed, if 
both fourth-generation quarks exist, we expect to select 
both modes in the £^£^jb^T sample. 

In general, the ratio of event yield to the integrated lu- 
minosity N/L equals the cross section multiplied by the 
acceptance rate. For a particular process, such as an indi- 
vidual b' , this gives the limit on cross section: ah' = t-^ 
where ef is the acceptance rate for the observed process 
within the experimental selection constraints. However, 
we can also consider the case with two contributions (i.e.. 



from t' and b') if we know the relative acceptance rates 
between the corresponding processes, Sreh and if the two 
cross sections are dependent: 

N 



(L-e,,) 



■ <^b' +^rel ■ O-t'i^'b')- 



(1) 



The t' and b' cross sections, nearly the same up 
to the t'- b' mass difference, are both determined by 
perturbative QCD. (Electroweak production modes of 
pairs of the fourth generation quarks are expected to 
be < 1% at the Tevatron). This is parameterized from 
the ncxt-to-leading order cross-section calculations for 
strong force production of massive quarks [11], by fitting 
the published theoretical cross-section to an exponential 
model in the local region of mass under consideration: 
at' = at' exp — "'''j^™'*' , where M is found to be 40 GeV. 

This relationship yields cross-section limits and, by ex- 
tension, mass limits on b' for a spectrum of assumed t' 
masses. The relative efficiency Crei between the origi- 
nal model and the model explicitly including both t' and 
b' was estimated using simulated data. The model for 
erei accounts for increased acceptance of t' as the fourth- 
generation mass difference increases, with a plateau be- 
yond mw- This method was validated by varying the 
model for erei- The resulting mass limits were found to 
be stable. We find that the additional contribution from 
the t' — > Wb' decay gives the limits mb' > 340 — 360 
depending on mf (Fig. 2a), stronger than the previous 
CDF individual b' limit. 

We probe the full two-dimensional branching fraction 
interval by calculating the dependence of the event yield 
on the branching fractions explicitly. As the branch- 
ing fractions to the reconstructed states vary, the accep- 
tances and reconstruction efficiencies of the processes of 
interest vary accordingly. Considering these effects, we 
calculate the acceptances of b' and t' in both the £ + 4j 
and l^i^jb^T samples. 

The expected signal yield for one process relative to 
another is proportional to the relative production rates 
and final-state reconstruction eflficiencies. The relative 
signal production rates have two factors: the relative 
cross section for initial state production (i.e., t'P and 
b'b') and the branching ratios of the involved processes. 
For several processes contributing to a signal there are 
multiple terms of this form. With the event yield fixed 
at its observed value, we isolate the cross section, and 
express it in terms of the previously measured limit and 
an effective relative acceptance. The eff'ective relative ac- 
ceptance (A) includes acceptance terms for all processes 
considered, each scaled by the relative cross-section. 

For the £+4:j sample considered with the classical split- 
ting case rrit' > mi,' , there are no relative reconstruction 
efficiencies to consider so the expression for the relative 
expected yields as a function of /3(,' = B{b' -^ Wt) and 
jSf = B{t' -^ Wb') is fairly simple: 



on the mass of the b' as a function of fourth-generation 
branching fractions. 

The i^i^jb^T case is somewhat more complicated. In 
addition to 6', there are two significant t' processes that 
produce the signal selected for the sample analyzed: 

t' -> Wb' -^ WWt ^ WWWb , 
t' -> Wb' ^ WWc . 

Jet multiplicity and the flavor-tag requirement imply 
different relative reconstruction efficiencies for each con- 
tribution. These factors are denoted e^Wy where N is 
the number of intermediate VF-bosons in the process de- 
scribed. They are estimated from simulated data and are 
calculated relative to the four-VF case considered in the 
original analysis. The effective relative acceptance in this 
case has the form 

A(/3,,,A0-/3,2.+/3,2,^[(l-/3,.^cc 
efcb 

+ 2f3i,'{l ~ (3b')e5w^cb + /^b'^ew^bb] , (3) 

where the individual efficiencies for each combination of 
decay modes are estimated by producing pairs of mas- 
sive quarks at tree level using the MadEvent software 
package [12] for a user-defined model containing t' and b' 
and allowing for O ~ 1% mixing with either the third or 
second generations (i.e., V4J ~ 0(0. 1) for j — 2,3). Un- 
der these assumptions, all of the inclusive decay widths 
(including t' -^ Wb' which is unsuppressed by mixing) 
are well within the expected jet energy resolution at a 
hadron collider. The events are generated at b' masses of 
400 GeV and for two different choices of mass splitting, 
50 GeV and 100 GeV, representing both on-shcU and off- 
shell W bosons in the t' -^ b' decay. The events arc pro- 
cessed into the desired decays modes using the BRIDGE 
program [13]. As expected, in the region of interest the 
dependence of efficiencies on quark-mass choices is small. 
Terms from the t' contribution have unique reconstruc- 
tion efficiencies due to jet-flavor tagging, expressed in 
eq. 3 by e/1/2, where /1/2 is the flavor combination of 
the quarks in the final state of the (quark-level) t' decay. 
These efficiencies are given by statistics from the raw ef- 
ficiencies of the jet-flavor tag to select the beauty (60%) 
or charm (15%) flavor in a jet: 

ebb = l-(l-efc)2 

^cb = ec(l - £&) + £&(! - Cc) + efeCc 

ecc = 1 - (1 - ec)' . 



(4) 



A{l3b',l3t') = (1 - Pb'f + ^rel{l - Pt'f 



(2) 



where ct^ 



CTj/cr(,. This treatment produces limits 



Using the above procedure, we produce limits on the 
mass of the b' as a function of fourth-generation branch- 
ing fractions. 

The two sets of results reveal the complementary sen- 
sitivities of the CDF analyses over the interval. However, 
they arc not orthogonal and thus cannot be statistically 
combined. Instead, the best limits are found by choosing 
the stronger of two limits at each point in the branching 
fraction interval. The combined results are presented for 
two characteristic choices of mass splitting in Fig. 2b, c. 



Finally, we consider the case of an inverted splitting, 
nihi > nit' ■ The CDF £ + Aj data sets a limit on t' for 
every choice of branching fraction in this case, as its as- 
sumption that B{t' — !■ Wq = d, s, b) — 100% is always 
satisfied. If B{b' — > Wq = u,c) is significant it also con- 
tributes, but at a smaller cross-section due to its larger 
mass. With the roles of b' and t' reversed, arei is in- 
verted accordingly in this case. The relative acceptance 
as a function of /3(,' = B{b' -^ Wt') is given by: 



A{Pv) =l+arelh'^ 



(5) 



On the other hand The i^i^jb^r sample, sensitive to 
6-type quarks, cannot at its present size set limits in the 
inverted splitting case, as the limits it produces would 
imply a lighter t' already excluded by the £ + 4j sample. 

Our limits for the inverted case to;,' > rrit' with the 
£ + Aj sample are shown in Fig. 2d. The displayed dis- 
continuity in the upper curve of Fig. 2d is caused by the 
corresponding slope in the £ -\- 4j result. 

To summarize, by analyzing the expectations for rel- 
ative event yields at CDF under the combined t',6' hy- 
pothesis, and assuming a continuum of t' and b' branch- 
ing fractions, we find that the CDF data imply limits on 
rub' and mt' of 290 GeV and greater over the full range 
of mixing scenarios, for two characteristic choices of the 



t'- b' mass splitting: mt' > mi' and irib' > mt'. The 
inclusion of a t' strengthens the previously obtained b' 
mass limit from the £ £ jb^T sample; in the nit' > mi,' 
case, by up to 10% when < mt' — rrib' < Mw- 

If a fourth generation of fermions is embedded in 
theories beyond the SM, then the large splitting case 
(mt' — TOfe' > Mw) and the inverted scenario (to.&' > vnc) 
may not be excluded. Indeed, an example given recently 
in [7] shows that precision EW data can accommodate 
TOi' — m.b' > Mw if there are two Higgs doublets. In fact, 
the compositeness picture emerging from the addition of 
new heavy fermionic degrees of freedom is more naturally 
described at low energies by multi- Higgs theories [5-7], 
for which constraints on the fourth generation parameter 
space are known to be relaxed [10]. In such cases (i.e., a 
large t'- b' mass splitting and/or an inverted splitting), 
we find new stronger bounds on m^' and ttic, in some 
mixing scenarios reaching up to 430 GeV. 
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